{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Datawhale 智慧海洋建设-Task2 数据分析"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"此部分为智慧海洋建设竞赛的数据分析模块，通过数据分析，可以熟悉数据，为后面的特征工程做准备，欢迎大家后续多多交流。\n",
"\n",
"赛题：智慧海洋建设"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"数据分析的目的:\n",
"\n",
"- EDA的主要价值在于熟悉整个数据集的基本情况(缺失值、异常值)，来确定所获得数据集可以用于接下来的机器学习或者深度学习使用。\n",
"\n",
"\n",
"- 了解特征之间的相关性、分布，以及特征与预测值之间的关系。\n",
"\n",
"\n",
"- 为进行特征工程提供理论依据。"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"项目地址：https://github.com/datawhalechina/team-learning-data-mining/tree/master/wisdomOcean\n",
"\n",
"\n",
"比赛地址：https://tianchi.aliyun.com/competition/entrance/231768/introduction?spm=5176.12281957.1004.8.4ac63eafE1rwsY"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2.1 学习目标"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"1. 学习如何对数据集整体概况进行分析，包括数据集的基本情况(缺失值、异常值)\n",
"\n",
"\n",
"2. 学习了解变量之间的相互关系、变量与预测值之间的存在关系。\n",
"\n",
"\n",
"3. 完成相应学习打卡任务"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2.2 内容介绍"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"1. 数据总体了解\n",
" - 读取数据集并了解数据集的大小，原始特征维度；\n",
" - 通过info了解数据类型；\n",
" - 粗略查看数据集中各特征的基本统计量\n",
"2. 缺失值和唯一值\n",
" - 查看数据缺失值情况\n",
" - 查看唯一值情况\n",
"3. 数据特性和特征分布\n",
" - 三类渔船轨迹的可视化\n",
" - 坐标序列可视化\n",
" - 三类渔船速度和方向序列可视化\n",
" - 三类渔船速度和方向的数据分布\n",
"4. 总结"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2.3 代码示例"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2.3.1 载入各种数据科学以及可视化库"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"以下库都是pip install安装，有特殊情况我会单独说明 例如 pip install pandas -i https://pypi.tuna.tsinghua.edu.cn/simple"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"ExecuteTime": {
"end_time": "2021-03-22T13:03:37.326461Z",
"start_time": "2021-03-22T13:03:36.094270Z"
}
},
"outputs": [],
"source": [
"#coding:utf-8\n",
"#导入warnings包，利用过滤器来实现忽略警告语句。\n",
"import warnings\n",
"warnings.filterwarnings('ignore')\n",
"\n",
"import numpy as np\n",
"import pandas as pd\n",
"from matplotlib import pyplot as plt\n",
"import seaborn as sns"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"df=pd.read_csv(r'D:\\WORK_DATA_F\\jupyter_notebook\\2020数字中国创新大赛-智慧海洋建设\\代码练习\\数据\\DF.csv')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2.3.2 载入其它相关的包"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"ExecuteTime": {
"end_time": "2021-03-22T13:03:37.340423Z",
"start_time": "2021-03-22T13:03:37.327459Z"
}
},
"outputs": [],
"source": [
"from tqdm import tqdm\n",
"import multiprocessing as mp\n",
"import os\n",
"import pickle\n",
"import random\n",
"# 把读取所有数据的函数放在单独的python文件中，是为了解决多线程问题在jupyter notebook无法运行的问题\n",
"import read_all_data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 说明："
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"本次数据分析探索，尤其可视化部分均选取某些特定变量进行了举例，所以它只是一个方法的展示而不是整个赛题数据分析的解决方案。"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2.3.3 定义加载和存储数据的函数"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"ExecuteTime": {
"end_time": "2021-03-22T13:03:37.982564Z",
"start_time": "2021-03-22T13:03:37.978574Z"
}
},
"outputs": [],
"source": [
"class Load_Save_Data():\n",
" def __init__(self,file_name=None):\n",
" self.filename = file_name\n",
"\n",
" def load_data(self,Path=None):\n",
" if Path is None:\n",
" assert self.filename is not None,\"Invalid Path....\"\n",
" else:\n",
" self.filename = Path\n",
" with open(self.filename,\"wb\") as f:\n",
" data = pickle.load(f)\n",
" return data\n",
"\n",
" def save_data(self,data,path):\n",
" if path is None:\n",
" assert self.filename is not None,\"Invalid path....\"\n",
" else:\n",
" self.filename = path\n",
" with open(self.filename,\"wb\") as f:\n",
" pickle.dump(data,f)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2.3.4 读取数据"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"ExecuteTime": {
"end_time": "2021-03-22T13:03:38.575284Z",
"start_time": "2021-03-22T13:03:38.571295Z"
}
},
"outputs": [],
"source": [
"# 定义读取数据的函数\n",
"def read_data(Path,Kind=\"\"):\n",
" \"\"\"\n",
" :param Path:待读取数据的存放路径\n",
" :param Kind:'train' of 'test'\n",
" \"\"\"\n",
"\n",
" # 替换成数据存放的路径\n",
" filenames = os.listdir(Path)\n",
" print(\"\\n@Read Data From\"+Path+\".........................\")\n",
" with mp.Pool(processes=mp.cpu_count()) as pool:\n",
" data_total = list(tqdm(pool.map(read_all_data.read_train_file if Kind == \"train\" else \n",
" read_all_data.read_test_file,filenames),total=len(filenames)))\n",
" print(\"\\n@End Read total Data............................\")\n",
" load_save = Load_Save_Data()\n",
" if Kind == \"train\":\n",
" load_save.save_data(data_total,\"./data_tmp/total_data.pkl\")\n",
" return data_total"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"ExecuteTime": {
"end_time": "2021-03-22T13:03:48.196318Z",
"start_time": "2021-03-22T13:03:38.955234Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"@Read Data FromD:/code_sea/data/train/hy_round1_train_20200102/.........................\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|█████████████████████████████████████████████████████████████████████████| 7000/7000 [00:00<00:00, 3527165.79it/s]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"@End Read total Data............................\n",
"\n",
"@Read Data FromD:/code_sea/data/test/hy_round1_testA_20200102/.........................\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|██████████████████████████████████████████████████████████████████████████████████████| 2000/2000 [00:00\n",
"Int64Index: 2699638 entries, 0 to 365\n",
"Data columns (total 7 columns):\n",
" # Column Non-Null Count Dtype \n",
"--- ------ -------------- ----- \n",
" 0 渔船ID 2699638 non-null int64 \n",
" 1 x 2699638 non-null float64\n",
" 2 y 2699638 non-null float64\n",
" 3 速度 2699638 non-null float64\n",
" 4 方向 2699638 non-null int64 \n",
" 5 time 2699638 non-null object \n",
" 6 type 2699638 non-null object \n",
"dtypes: float64(3), int64(2), object(2)\n",
"memory usage: 164.8+ MB\n"
]
}
],
"source": [
"pd.options.display.max_info_rows = 2699639\n",
"data_train.info()"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"ExecuteTime": {
"end_time": "2021-03-22T13:03:57.019335Z",
"start_time": "2021-03-22T13:03:56.490727Z"
},
"scrolled": false
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" 渔船ID | \n",
" x | \n",
" y | \n",
" 速度 | \n",
" 方向 | \n",
"
\n",
" \n",
" \n",
" \n",
" | count | \n",
" 2.699638e+06 | \n",
" 2.699638e+06 | \n",
" 2.699638e+06 | \n",
" 2.699638e+06 | \n",
" 2.699638e+06 | \n",
"
\n",
" \n",
" | mean | \n",
" 3.496035e+03 | \n",
" 6.277243e+06 | \n",
" 5.271190e+06 | \n",
" 1.784449e+00 | \n",
" 1.151533e+02 | \n",
"
\n",
" \n",
" | std | \n",
" 2.020781e+03 | \n",
" 2.698065e+05 | \n",
" 2.544160e+05 | \n",
" 2.478862e+00 | \n",
" 1.168515e+02 | \n",
"
\n",
" \n",
" | min | \n",
" 0.000000e+00 | \n",
" 5.000250e+06 | \n",
" 3.345433e+06 | \n",
" 0.000000e+00 | \n",
" 0.000000e+00 | \n",
"
\n",
" \n",
" | 1% | \n",
" 6.900000e+01 | \n",
" 5.258862e+06 | \n",
" 4.618927e+06 | \n",
" 0.000000e+00 | \n",
" 0.000000e+00 | \n",
"
\n",
" \n",
" | 2.5% | \n",
" 1.710000e+02 | \n",
" 5.817836e+06 | \n",
" 4.920685e+06 | \n",
" 0.000000e+00 | \n",
" 0.000000e+00 | \n",
"
\n",
" \n",
" | 5% | \n",
" 3.470000e+02 | \n",
" 6.024286e+06 | \n",
" 4.985102e+06 | \n",
" 0.000000e+00 | \n",
" 0.000000e+00 | \n",
"
\n",
" \n",
" | 50% | \n",
" 3.502000e+03 | \n",
" 6.246522e+06 | \n",
" 5.229463e+06 | \n",
" 3.200000e-01 | \n",
" 8.100000e+01 | \n",
"
\n",
" \n",
" | 75% | \n",
" 5.243000e+03 | \n",
" 6.365916e+06 | \n",
" 5.379267e+06 | \n",
" 3.290000e+00 | \n",
" 2.170000e+02 | \n",
"
\n",
" \n",
" | 90% | \n",
" 6.290000e+03 | \n",
" 6.592496e+06 | \n",
" 5.602273e+06 | \n",
" 4.910000e+00 | \n",
" 2.930000e+02 | \n",
"
\n",
" \n",
" | 99% | \n",
" 6.928000e+03 | \n",
" 7.056209e+06 | \n",
" 6.111650e+06 | \n",
" 1.009000e+01 | \n",
" 3.560000e+02 | \n",
"
\n",
" \n",
" | max | \n",
" 6.999000e+03 | \n",
" 7.133785e+06 | \n",
" 7.667581e+06 | \n",
" 1.001600e+02 | \n",
" 3.600000e+02 | \n",
"
\n",
" \n",
"
\n",
"
\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" 渔船ID | \n",
" x | \n",
" y | \n",
" 速度 | \n",
" 方向 | \n",
" time | \n",
" type | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" 0 | \n",
" 6.152038e+06 | \n",
" 5.124873e+06 | \n",
" 2.59 | \n",
" 102 | \n",
" 1110 11:58:19 | \n",
" 拖网 | \n",
"
\n",
" \n",
" | 1 | \n",
" 0 | \n",
" 6.151230e+06 | \n",
" 5.125218e+06 | \n",
" 2.70 | \n",
" 113 | \n",
" 1110 11:48:19 | \n",
" 拖网 | \n",
"
\n",
" \n",
" | 2 | \n",
" 0 | \n",
" 6.150421e+06 | \n",
" 5.125563e+06 | \n",
" 2.70 | \n",
" 116 | \n",
" 1110 11:38:19 | \n",
" 拖网 | \n",
"
\n",
" \n",
" | 363 | \n",
" 999 | \n",
" 6.138413e+06 | \n",
" 5.162715e+06 | \n",
" 0.32 | \n",
" 0 | \n",
" 1031 12:28:01 | \n",
" 拖网 | \n",
"
\n",
" \n",
" | 364 | \n",
" 999 | \n",
" 6.138413e+06 | \n",
" 5.162715e+06 | \n",
" 0.32 | \n",
" 0 | \n",
" 1031 12:18:00 | \n",
" 拖网 | \n",
"
\n",
" \n",
" | 365 | \n",
" 999 | \n",
" 6.138413e+06 | \n",
" 5.162715e+06 | \n",
" 0.11 | \n",
" 294 | \n",
" 1031 12:07:59 | \n",
" 拖网 | \n",
"
\n",
" \n",
"
\n",
"